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SYSTEM AND METHOD FOR VIDEO CONTENT ANALYSIS-BASED 
DETECTION, SURVEILLANCE, AND ALARM MANAGEMENT 

BACKGROUND OF THE INVENTION 
5 RELATED APPLICATIONS 

The present invention relates and claims priority from US provisional 
patent application serial number 60/354,209 titled ALARM SYSTEM BASED 
ON VIDEO ANALYSIS, filed 6 February 2002. 

0 FIELD OF THE INVENTION 

The present invention relates to video surveillance systems in general, 
and more particularly to a video content analysis-based detection, surveillance 
and alarm management system. 

15 DISCUSSION OF THE RELATED ART 

Due to the increasing number of terror attacks and potential terror- 
related threats, one of the most critical surveillance challenges today is the timely 
and accurate detection of suspicious objects, such as unattended luggage, illegally 
parked vehicles, suspicious persons, and the like, in or near airports, train stations, 

20 federal and state government buildings, hotels, schools, crowded public places 
typically situated at city centers, and other sensitive areas. In accordance with the 
prevailing known tactics of terrorist organizations, unattended innocent-looking 
objects, such as a suitcase, could contain hidden explosive materials installed 
therein to effect a controlled explosion and thereby inflict massive impact damage 

25 to the near environment and the individuals within. Likewise recent attacks have 
been perpetrated through the use of vehicle bombs, seemingly innocent until 
explosion. Since these hidden explosives are usually activated by the setting of 
carefully timed (typically short-period) detonator means, or by an operator who is 
at the scene or close by, the prompt, rapid and timely detection of suspicious 
30 objects, such as unattended luggage, vehicles parking in forbidden zones, 



1 



'^fc jfc PCT/IL02/01042 

WO 03/067360 

suspicious persons, persons leaving unattended suitcases or vehicles, and the like 
could prevent life-threatening situations. Similarly, it is important in areas, such 
as airports, to be able to track persons and objects, such as suitcases and cars, to 
assist in locating lost luggage, and to restrict access of persons or cars to certain 
5 zones. The applications of such abilities are not only for security purposes. 

Recently, the authorities responsible for the safety of the public have 
been attempting to cope with the problems listed above in the most obvious 
manner by increasing the number of human personnel tasked for the detection, 
identification and consequent handling of suspicious objects, including vehicles, 
10 luggage and persons. At the same time, in order to maintain substantially 
unobstructed passenger flow and in order to minimize transport delays and 
consequent public frustration, the security personnel have been obliged to utikze 
inefficient and time-consuming procedures. One drawback of the above human- 
centric solution concerns the substantially increased expenses associated with the 
15 hiring of a large number of additional personnel. Another drawback concerns the 
inherent inefficiency of the human-centric procedures involved For example, 
specific airport security personnel must perform visual scanning, tracking, and 
optional handling of objects in sensitive transit areas 24 hours a day, where the 
sheer number of luggage passing through these areas effect increased fatigue 
20 accompanied naturally with diminished concentration. In the same manner, in a 
traffic-extensive area wherein specific security personnel must watch, track and 
optionally handle vehicles parking in restricted areas, natural weariness soon sets 
in and the efficiency of the human-centric procedure gradually deteriorates. 

Currently available surveillance systems are designed for assisting 
25 human security officers. These systems typically include various image 
acquisition devices, such as video cameras, for capturing and recording imagery 
content, and various detector devices, such as movement detectors. The existing 
surveillance systems have several important disadvantages. The type of alarms 
provided by the detectors is substantially limited. The video images recorded by 
30 the cameras are required to be monitored constantly by human security personnel 
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in order to detect suspicious objects, people and behavior. An alarm situation has 
to be identified and suitably handled by the personnel where a typical handling 
activity is the manual generation and distribution of a suitable alarm signal. Since 
these surveillance systems are based on human intervention the problems related 
5 to natural human-specific processes, such as fatigue, lack of concentration, and 
the like, are still remain in effect 

A further drawback of existing surveillance systems concern the 
failure of those systems to handle certain inherently suspicious events that were 
captured by the cameras monitoring a scene. For example, current surveillance 
10 . systems associated with airport security application, typically fail to identify a 
situation as suspicious where the situation involves a vehicle arriving at a 
monitored airport terminal, an occupant of the vehicle leaving of the vehicle, and 
the departure of the occupant from the monitored scene in a direction that is 
opposite to the terminal. 
15 Yet another drawback of the current systems concern the inability of 

the current systems to identify a set of events linked to the same object in the 
same area throughout a pre-defined surveillance period. For example, when 
"suitcase" object is left in the scene by a first person and later it is picked up by a 
second person then the leaving of the suitcase and the picking up of the suitcase 
20 constitute a set of linked events. 

Still another drawback of the current systems concern the inherent 
passivity of systems due to the fact that the operations of the systems are based on 
events initiated by the operators and due to the fact that the systems provide no 
built-in alerts. 

25 In addition, existing systems are incapable of associating a retrieved 

event or object through the use of important parameters, such as color of hair, 
color of clothing and shoes, complexion (via the use of a color histogram), facial 
features (via face recognition routines), normalized size of the object (distance 
from the camera), and the like. 



3 



WO 03/067360 




PCT/IL02/01042 



It would be easily perceived by one with ordinary skills in the art that 
there is a need for an advanced and enhanced surveillance, object tracking and 
identification system. Such a system would preferably automate the procedure 
concerning the identification of an unattended object substantially and would 
5 utilize cost-effective, efficient methods. 

SUMMARY OF THE PRESENT INVENTION 
One aspect of the present invention relates to a method for analyzing video 
data, comprising receiving a video frame, comparing said video frame to 
10 background reference frame to locate difference, locating a plurality of objects to 
form a plurality of marked objects; and detennining a behavior pattern for an 
object according to the difference, said behavior pattern is defined according at 
least one scene characteristic. The method further comprises producing an 
updated background reference frame. The method further comprising 
15 determining the difference performed by creating a difference frame between the 
video frame and the background reference frame. The method further comprises 
finding a new object when determining the difference and an alarm according to 
said behavior pattern. A pre-defined pattern of suspicious behavior comprises an 
object presenting unpredictable behavior. 
20 A second aspect of toe present invention relates to a system for analyzing 

video data comprising a plurality of video frames, the system comprising, a video 
frame preprocessing layer for determining a difference between a plurality of 
video frames, an object clustering layer for detecting a plurality of objects 
according to said difference, and an application layer for characterizing said 
25 plurality of objects according to scene characteristic. The difference is 
determined between a video frame and a reference frame. The system further 
comprises a background refreshing layer for preparing an updated reference 
frame according to toe said difference. The scene characteristic defines a 
behavior pattern for an object, such that if the object exhibits the behavior 
30 pattern, the scene characteristic is detected. If the scene characteristic is detected, 
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an alarm is generated. The scene characteristic further comprises a parameter for 
determining if the object exhibits the behavior pattern. 

A third aspect of the present invention refers to a system for detecting a 
vehicle remaining in a restricted zone for at least a minim um period of time, 
5 comprising, a video content analysis module for analyzing video data of the 
restricted zone, said video content analysis module further comprising an object 
tracking component, and an application layer for receiving data from said video 
content analysis module and for detecting a vehicle remaining in the restricted 
zone for the niinimum period of time, and said application layer generating an 
10 alarm upon detection. 

A fourth aspect of the present invention refers to a system for detecting 
unattended luggage, bag or any unattended object in an area, comprising, a video 
content analysis module for analyzing video data of the area, said video content 
analysis module further comprising an object tracking component, and an 
15 application layer for receiving data from the video content analysis module and 
for detecting an unattended object, wherein said unattended object has not been 
attended in the area for more then a predefined period of time. 

A fifth aspect of the present invention refers to a surveillance system for the 
detection of an alarm situation, the system comprising the elements of, a video 
20 analysis unit for analyzing video data representing images of a monitored area, 
the video analysis unit comprising an object tracking module to track the 
movements and the location of a video object, a detection, surveillance and alarm 
application for receiving video data analysis results from the video analysis unit, 
for identifying an alarm situation and to generate an alarm signal, an events 
25 database to hold video objects, video object parameters and events identified by 
the application. The system comprises the elements of; an application driver to 
control the detection, surveillance and alarm application, a database handler to 
access, to update and to. read the events database, a user interface component to 
communicate with a user of the system, an application setup and control 
30 component to define the control parameters of the application, an application 
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setup parameters table to store the control parameters of the application. The 
system further comprises the elements of, a video data recording and 
compression unit to record and compress video data representing images of a 
monitored area, a video archive file to hold the recorded and compressed video 
data representing images of the a monitored area, an alarm distribution unit to 
distribute the alarm signal representing an alarm situation. The system further 
comprises the elements of, a video camera to obtain the images of a monitored 
area, a video capture component to capture video data representative of the 
images of the monitored area, a video transfer component to transfer the captured 
video data to the video analysis unit and the recording compressing and archiving 
unit, a computing and storage device. The object tracking module comprises the 
elements of, a video frame preprocessing layer for deternfining the difference 
between video frames, an objects clustering layer for detecting objects in 
accordance with the determined difference, a scene characterization layer for 
characterizing the object according to characteristic of a scene, a background 
refreshing layer for preparing an updated reference according to the determined 
difference. The detection surveillance and alarm application is operative in the 
detection of an unattended object in the monitored area. Any video camera within 
the system, the video capturing component, the video transfer component and the 
computing and storage device can be separated and can be located in different 
locations. The interface between the video camera, the video capturing 
component, the video transfer component and the computing and storage device 
is a local or wide area network or a packed-based or cellular or radio frequency 
or micro wave or satellite network. The unattended object is a luggage left in an 
airport terminal for a pre-deterrnined period or a vehicle parking in a restricted 
zone for a pre-defined period. The detection surveillance and alarm application is 
operative in the detection of an unpredicted object movement The analysis is 
also performed on audio data or thermal imaging data or radio frequency data 
associated with the video data or the video object in synchronization with the 
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video data. The video capture component captures audio or thermal information 
or radio frequency information in synchronization with the video data. 

A sixth aspect of the present invention refers to a surveillance method for 
the detection of an alarm situation, the surveillance to be performed on a 
5 monitored scene having a camera, the method comprising the steps of, obtaining 
video data from the camera representing images of a monitored scene, analyzing 
the obtained video data representing images of the object within the monitored 
scene, the analyzing step comprising of identifying the object within the video 
data, and inserting the identified object and the event into an event database. 
10 Another embodiment of the method further comprises the steps of, retrieving of 
the object associated with an event, according to user instruction displaying the 
video event associated with the retrieved object The method further comprises 
the steps of; retrieving at least two events, associating according to parameters of 
the object, the object with the at least two events. The method comprises the 
15 steps of, debriefing the object associated with the event to identify the pattern of 
behavior or movement of the object within the scene within a predefined period 
of time. The method further comprises the steps of, pre-defining patterns of 
suspicious behavior, and pre-defining control parameters. The method further 
comprises the steps of; recognizing an alarm situation according to the pre- 
20 defined patterns of suspicious behavior, and generating an alarm signal 
associated with the recognized alarm situation. The method further comprises the 
steps of; implementing patterns of suspicious behavior introducing pre-defined 
control parameters, recording, compressing and archiving the obtained video 
data, distributing the alarm signal representing an alarm situation across a pre- 
25 defined range of user devices. The pre-defined pattern of suspicious behavior 
comprises, an object entering a monitored scene, the object separating into a first 
distinct object and a second distinct object in the monitored scene, the first 
distinct object remaining in the monitored scene without movement for a pre- 
defined period, and the second distinct object leaving the monitored scene! The 
30 pre-defined pattern of suspicious behavior comprises, an object entering the 
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monitored scene, the object ceasing its movement, the size of the object is 
recognized as being above a pre-defined parameter value, and the object 
remaining immobile for a period recognized as being above a pre-defined 
parameter value. The method does further comprise identifying information 
5 associated wim toe object for me pu^ 

BRIEF DESCRIPTION OF THE DRAWINGS 
The present invention will be understood and appreciated more fully 
from me following detailed description taken in conjunction with the drawings in 
10 which: 

Fig. 1 is a schematic block diagram of the system architecture, in 
accordance with the preferred embodiments of the present invention; 

Fig. .2 is a simplified flowchart that illustrates the operation of the 
object tracking method, in accordance with the preferred embodiment of the 

15 present invention; 

Fig. 3A is a simplified flowchart illustrative of the operation of 
unattended luggage detection application, in accordance wim the first preferred 
embodiment of the present invention; 

Fig. 3B illustrates the control parameters of the unattended luggage 
20 detection application, in accordance with the first preferred embodiment of the 
present invention; 

Fig. 4A is a simplified flowchart Mustrative of the operation of the city 
center application, in accordance with me second preferred embodiment of the 

present invention; 

Fig. 4B illustrates the control parameters of the city center application, 
in accordance with the second preferred embodiment of the present invention; and 
Fig. 5 is a flowchart describing the operation of the proposed method, 
in accordance with the preferred embodiments of the present, invention. 
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DETAILED DESCRIPTION OF TEE PREFERRED EMBODIMENT 

A monitoring and surveillance system and method for the detection of 
potential alarm situation via a recorded surveillance content analysis and for the 
management of the detected unattended object situation via an alarm distribution 
5 mechanism is disclosed. The proposed system and method includes an advanced 
architecture and a novel technology operative in capturing surveillance content, 
analyzing the captured content and providing in real time a set of alarm messages 
to a set of diverse devices. The analysis of the captured content comprises a 
unique algorithm to detect, to count and to track objects embedded in the captured 
10 content. The present invention provides a detailed description of the applications 
of this method. The method and system of the present invention may be 
implemented in the context of unattended objects (such as luggage, vehicles or. 
persons), parking or driving in restricted zones, controlling access of persons into 
restricted zones, preventing loss of objects such as luggage or persons and 

15 counting of persons. 

In me preferred embodiments of the present invention, the monitored 
content is a video stream recorded by video cameras, captured and sampled by a 
video capture device and transferred to a video processing unit The video 
processing unit performs a content analysis of the video images and indicates an 
20 alarm situation in accordance with the results of the analysis. In other preferred 
embodiment of the invention, diverse other content formats are also analyzed, 
such as thermal based sensor cameras, audio, wireless linked camera, data 
produced from motion detectors, and the like. 

25 The first preferred embodiment of the present invention concerns the 

detection of unattended objects, such as luggage in a dynamic object-rich 
environment, such as an airport or city center. The second preferred embodiment 
of the invention concerns the detection of a vehicle parked in a forbidden zone, or 
the extended-period presence of a non-moving vehicle in a restricted-period 

30 parking zone. Forbidden or restricted parking zones are typically associated with 
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sensitive traffic-intensive locations, such as a city center. Another preferred 
embodiment of the invention concerns the tracking of objects such as persons m 
various scenarios, such as a person leaving the vehicle away from the termmal, 
which may equal suspicious (unpredicted) behavioral pattern. In other posstble 

5 embodiments of the present mvention the system and method can be implemented 
to assist m locating lost luggage and to restrict access of persons or vehicles to 
certain zones. Other preferred embodiments of the invention could regard me 
detection of diverse other objects in diverse other environments. The followmg 
description is not meant to be limiting and the scope of the mvention is defined 

10 only by the attached claims. 

Referring to Fig. 1 a set of video cameras 12, 14, 16, 18 operate in a 
security-wise sensitive environment and cover a specific pre-defined zone that is 
required to be monitored. The area monitored can be any area preferably in a 
15 transportation area including an airport, a city center, a building, and restricted or 
non-restricted, areas within buildings or outdoors. The cameras 12, 14, 16, 18 
could be analog devices or digital devices. The cameras can capture normal fight, 
infra-red, temperature, or any other form of radiation. By using audio capturing 
devices such as a microphone (not shown), me cameras can also capture auditory 
signals, such as noise generated by machines and voices made by persons. The 
cameras 12, 14, 16, 18 continuously acquire and transmit sequences of video 
images to a display device 21, such as a video terminal, operated by a human 
operator. The display device 21 could be optionally provided with video images 
from the vide archives 40 by the computing and storage device 24. The cameras, 
12, 14, 16, 18 transmit sequences of video images to a video capture component 
20 via'suitLbly wired connections. The video capture component 20 could capture 
the images through an analog interface, a digital interface or through a Local Area 
Network (LAN) interface or Wide Area Network (WAN), IP, Wireless, Satellite 
connectivity. The video capture component 20 can be a NICEV1SION system 
manufactured by Nice Systems Ltd., Raanana, Israel. The video capture 
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component 20 can also be configured to capture audio signals captured by the 
. cameras audio capturing devices. The audio and video signals are preferably 
synchronized at the system level. The component 20 receives the sequences of 
video images and appropriately samples the video stream. Where the processing 
5 of the captured video stream is performed by an external computing platform, 
such as a Personal Computer (PC), a UNIX workstation, or a mainframe 
computer, the unit 20 sends the sampled video information to a video transfer 
component 22. The video transfer component 22 transfers the video information 
to a computing and storage device 24. The device 24 could be an external 
10 computing platform, such as a personal computer (PC), a UNIX workstation or a 
mainframe computer having appropriate processing and storage units or a 
dedicated hardware such as a DSP based platform. It is contemplated that future 
hand held devices will be powerful enough to also implement device 24 there 
within. The device 24 could be also an array of integrated circuits with built-in 
15 digital signal processing (DSP) and storage capabilities attached directly to the 
video capture component 20. The capture component 20 and the transfer 
component 22 are preferably separate due to the fact that a capture component 
can be located at the monitored scene, while a transfer component can be located 
away from the monitored scene. In another preferred embodiment the capture and 
20 the transfer components can be located in the same device. The device 24 
includes a video analysis unit 26, an application driver 30, a database handler 32, 
a user interface 34, a setup and control Detection Surveillance and Alarm (DSA) 
application 36, a recording compressing and archiving unit 38, a video archives 
40, an events database 42, an alarm distribution unit 44, a DSA application and 
25 setup parameters file 46, and DSA application 48. Optionally, whenever video is 
captured and processed, audio signals captured in association with these captured 
video signals can be stored and tagged as relating to the video captured and 



Still referring to Fig. 1 the video content is transferred optionally to the 
30 recording, compressing, and archiving unit 38. The unit 38 optionally compresses 
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the video-content and stores the compressed content to the video archive files 40. 
The video archive files 40 could be suitable auxiliary storage devices, such as 
optical disks, magnetic disks, magnetic tapes, or the like. The stored content is 
held on the file 40 for a pre-defined (typically long) period of time in order to 
5 enable re-play, historical analysis, and the like. In parallel the video content is 
transferred to the video analysis unit 26. The unit 26 receives the video input, 
activates the object tracking module 28 and activates the application driver 30. In 
accordance, with results of the video analysis performed in conjunction with the 
object tracking module 28, the video analysis unit 26 further generates the 
10 appropriate alarm or indication signal where a specific alarm situation is detected. 
The application driver 30 includes the logic module of the application 48. The 
driver 30 receives event data and alarm data from the video analysis unit 26 and 
inserts the event data and the alarm data via the database handler 32 into the 
events database 42. The driver 30 further controls the operation of the DSA 
15 application 48. The setup and control DSA application 36 is used by the user of 
the system in order perform system setup, to define control parameters, and the 
like. The user interface 34 is responsible for the communication with the user. 
The event database 42 stores the event data and the alarm data generated by the 
video analysis unit 26. The event database 42 also holds the search parameters for 
20 searching objects or events for the purpose of investigating the events or objects. 
The search parameters include the object circle-like shape and object location 
parameters. Other object search parameters can also include data collected from 
various cameras, which may have captured the same object The collected data 
could provide important information about the object, such as object type 
25 (animate or inanimate), object identification (via face recognition), color 
histogram (color of hair, of cloth, of shoes, of complexion), and the like. The 
parameters allow finding associations between objects and events captured by 
different cameras. The alarm distribution unit 44 optionally distributes the 
received alarm signals to a variety of alarm and messaging device. The DSA 
30 application and setup parameter ' file 46 stores the setup information and 
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parameters generated by the user via the setup and control DSA application 36. 
The DSA application 48 provides real-time video to the user, performs re-plays of 
video by request, submits queries to the event database 42 and provides alarm 
messages, such as suitably structured pop-up windows, to the nser via the nser 
interface 34. 

Still referring to Fig. 1 the units and components described could be 
installed in distinct devices distributed randomly across a Local Area Network 
(LAN) that could communicate over the LAN infrastructure or across Wide Area 
Networks (WAN). One example is a Radio Frequency Camera that transmits 
composite video remotely to a receiving station, the receiving station can be 
connected to other components of the system via a network or directly. The units 
and components described could be installed in distinct devices distributed 
randomly across very wide area networks such as the Internet Various means of 
communication between the constituent parts of the system can be used. Such can 
> be a data communication network, which can be connected via landlines or 
cellular or like communication devices and that can be implemented via TCP/IP 
protocols and like protocols. Other protocols and methods of communications, 
such as cellular, satellite, low band, and high band communications networks and 
devices will readily be useful in the implementation of the present invention. The 
0 components could be further co-located on the same computing platform or 
distributed across several platforms for load balancing. The components could be 
redundantly replicated across several computing platforms for specific 
operational purposes, such as being used as back-up systems in the case of 
equipment failure, and the like. Although on the drawing under discussion only a 
limited set of cameras and only a single computing and storage device are shown 
it will be readily perceived that in a realistic environment a plurality of cameras 
could be connected to a plurality of computing and storage devices. 

Referring now to Fig. 2 showing illustrates the operation of the object 
tracking method. The proposed system and method is based upon a video content 
analysis method that can detect, track and count objects in real time in accordance 
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*Ul the results of the video stream processing. This method enables the detection 
of new objeete-ereated ton the objeet identified and .racked The method also 
enables to identify when an objeet tracked has merged with another objeet The 
merging of objects with other objects and the creation of objects from other 
5 objects is particularly important to identify persons leaving vehicles, luggage, or 
other objeots in the environment monitored. The ability to detect if an object is 
created or disappears also enables the method of the present invention to identify 
if persons disturb objects or move objects. The method is implemented in the 
object tracking module 28 of Fig. 1, The method receives the following input 
10 new video frame, background (reference fame), detected object from the las. 
iteration. The ou*uts of the method comprise the updated background and the 
updated objects. Fig. 2 illustrates the four layers mat jointly implement object 
Backing and detection method The video ftame pre-processing layer 52 uses a 
, new ftame and one or more reference ftame for generating a difference ftame 
15 representing the difference between the new frame and the reference ftame or 
fl^es. The reference ftame can be obtained from, one of the capture devroes 
described in association with Fig. 1 or provided by the user. The difference ftame 
can be filtered or smootiened The objects clustering layer 54 generates 
new/updated objects ftom the difference ftame and the last known objects. The 
20 scene characterization layer 56 uses the objects ftom the objects clustering layer 
54 in order to describe the scene. The background-refteshing layer 58 updates the 
background (reference ftame or ftsmes) for the next frame calculation end a 
refreshing process uses tie outputs of all tie previous layers to generate a new 
reference layer or layers. Note should be taken fiat in other preferred 
25 embodiments of the invention other similar or different processes could be used 
to accomplish the underlying objectives of tie system and method proposed by 

the present invention. 

The first preferred embodiment of the invention regards an unattended 
object detection system and method. The unattended object could be a suitcase, a 
30 carrier bag, a backpack, or any other object that was left unattended in a security- 
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sensitive area, such as an airport terminal, a train station's waiting room, a public 

bunding, or the like. 

Referring now to Fig. 3A showing a flowchart mustrative of the 
operation of unattended luggage detection application of the method of the 
present invention. An unattended luggage event is detected via the performance 
of a sequence of operative steps. The logic underlying the performance is based 
on a specific scenario. In me scenario it is assumed that a terrorist or any other 
individual having criminal intent may enter a security-sensitive area carrying a 
suitcase. In one example, the suitcase may contain a concealed explosive device. 
!n another example, the suitcase may have been lost or unattended for a 
prolonged length of time. In yet another application, a suitcase or an object may 
have been taken without authority. In the first example, the individual may 
surreptitiously (in a manner that non-recognizable by the monitoring cameras) 
activate a time-delay fuse mechanism connected to the explosive device that * 
operative in the timed detonation of the device. Subsequently, the individual may 
abandon the suitcase unattended and leave the security-sensitive area for his own 
safety in order not to be exposed to the damage effected by the expected 
detonation of the explosive device. The following operative conclusions, 
indicating certain sub-events, are reached by the suitable execution of sets of 
computer instructions embedded within a specifically developed computer 
program. The program is operative in the analysis of a sequence of video images 
received from a video camera covering a security-sensitive area, referred herein 
below to as the video scene. An unattended luggage event is identified by the 
program when the following sequence of sub-events takes place and detected by 
the execution of the program: a) an object enters the video scene (62). It is 
assumed that the object is a combined object comprising an individual and a 
suitcase where the individual carries the suitcase, b) The combined object is 
separated into a first separate object and a second separate object (64). It is 
assumed that the individual (second object) leaves the suitcase (first object) on 
1he floor, a bench, or the like, c) The first object remains in the video scene 
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wit hout movement for a pre-defined period of time (66). It is assumed that the 
suitcase (first object) was left unattended, d) The second object exits the vxdeo 
scene (68) It is assumed that the individual (second object) left the video scene 
without the suitcase (first object) and is now about leave the wider area around 
the video scene. Following me identification of me previous sub-events, referred 
to collectively as the video scene characteristics, the event will be identified by 
the system as a situation in which an unattended suitcase was left in the secunty- 
sensitive area. Thus, the unattended suitcase will be considered as a suspicious 
object. Consequently, the proposed system may generate, display and/or 
distribute an alarm indication. Likewise, in an alternative embodiment in step 62, 
a first object, such as a suitcase or person monitored is already present and 
monitored within the video scene. Such object can be lost luggage located within 
the airport Such object can be a person monitored. In step 64 the object merges 
into a second object The second object can be a person picking up the luggage, 
another person to whom the first person joins or a vehicle to which the first 
person enters. In step 66 the first object (now merged with the second object) 
moves from its original position and. at step 68 of the alternative embodiment 
exists the scene. The system of the present invention will provide an indication to 
a human operator. The indication may be oral, visual or written. The indicatxon 
may be provided visually to a screen or delivered via communication networks to 
officers located at the scene or to off-premises or via dry contact to an external 
device such as a siren, a bell, a flashing or revolving light and the like. 

Referring now to Fig. 3A which illustrates the control parameters of 
the unattended luggage detection application. In order to set up the unattended 
5 luggage detection application, the user is provided with the capability of defimng 
me following control parameters: a) area or areas within the scanned zone 
wherein the system will search for suspected objects (70), b) the dimensional 
limits of the detected object (72). Objects having dimensions out of the limits 
defined will not be detected as suspected objects, and c) a time out value that is 
the amount of time that should pass from the point-in-time at which the suspected 
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object was detected as non-moving and the point-in-time until an alarm is 
generated. Once an alarm is raised the officer reviewing the monitored scene may 
request the system to provide a playback so as to identify the objects in question. 
Once playback resumes the officer may tap on a touch sensitive screen (or select 
5 the image by other means such as a mouse, a keyboard, a light pen and the hke) 
and the system may play back the history of video captured in association with 
the relevant object or objects. If a second object, such as a luggage left unattended 
is played back, the playback will identify the person taking or leaving the object. 
The officer may select such person and request play back or forward play to 
10 ascertain where the person came from or where the person went to and 
appropriately alert security officers. In order to debrief the event the officer may 
mine the database in various ways. One example would be to request the system 
to retrieve the events or objects that are similar to search parameters associated 
with the object or event he is . investigating that can help in identifying the 
15 location of person or objects or the whereabouts or actions performed by the 
object A suspect may place a suitcase given to him previously (not in the same 
scene) by a third person and leave the airport in a vehicle. Once the officer is 
alerted to the fact that the suitcase is unattended he may investigate the retrieve 
the third party associated with the handing of the suitcase and the vehicle 
20 associated with the suspect The system, in real time, stores from each camera the 
various objects viewed. In the ordinary course of events the system would 
associate between like objects captured by various cameras using the initial 
search parameter (such as the suspect's parameters). During the investigation 
stage the system retrieves the associated objects and the event linked therewith 
25 and presents the events and object to the viewing officer in accordance with his 
instructions. The officer may decide to review forward or backward in time 
scenes the system would mark the associated objects thus allowing the officer to 
identify the stream of events elected for a particular object, such as existing a 
vehicle, handing over a suitcase, leaving a suitcase unattended, walking in 
30 unpredicted directions and the like. 



17 



WO 03/067360 





|PCT/DU)2/01042 



As noted above the user may provide a predefined background. The 
backbond ,nay be eaptured to me eapturmg devices. The human operator 
may define elements within the screen as background elements. Such can be 
moving shades or areas of little interest and the like. 
5 Note should be taken that the above-described steps for the detection 

of a suspected object and the associated control parameters are exemplary only 
Diverse other sequences of steps and different control parameters could be user. 
' in order to achieve the inherent objectives of the present invenrion. 

The second preferred embodiment of the invention regards a detection 
,0 of vehicles parked in restricted area or moving in restricted lanes. Airports, 
government buildings, hotels and outer institutions typicaUy forbid vehicles from 
parking in specific arena or driving in reacted lanes. In some areas parking . 
forbidden all the time while in other areas parking is allowed for a short period, 
such as several minutes. In the second preferred embodiment of the invention a. 
,5 system and method is proposed mat detect vehicles paridng in restricted arena for 
more than a pre-defined number of time units and generates an alarm when 
idCTtayi n g an illegal parking event of a specific vehicle. In another preferred 
embodiment me system and memod of the present invention can detect whether 
persons disembark or embark a vehicle in predefined restricted zones. The use of 
20 the embodiment described in association with Figs. 3A, 3B can be employee in 
association with the application of the invention described below in association 

with Figs. 4A.4B. 

Referring now to Fig. 4A which shows ah exemplary flowchart 
mustrative of the operation of the chy center application An illegal parking event 
25 is detected via the performance of a sequence of operative ships. The logtc 
underlying the performance is based on a specific exemplary scenario. In me 
scenario it is assumed that one or more persons may drive into a security-sensmve 
area in a specific vehicle. The vehicle could contain a powerful hidden explosive 
device mat designed to be activated by one of me occupants of the vehicle or 
p^aons later embarking said vehicle in a restricted zone. Alternatively, the one or 
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more occupants of the vehicle could be armed and may plan en armed attack 
against specific targets, such as individuals entering or exiting the bmldmg On 
the ft* case scenario one of the occupants may surreptitiously (in amanner ttrat 
h ncn-recostizable by me monitoring cameras) activate a time-delay fuse 
m echanism connected to the explosive device that is operative in the rimed 
detonation of the device. Subsequent*, me occupants may abandon the vehtcle 
md le ave the seouriV-sensiuve area for their own safety in order not «o be 
exposed to me damage effected by the expected detonation of the explore 
device. In a seoond case scenario, the occupants may remain in the vehicle whtle 
waiting for the potential target, sueh as an individual about to enter the scene 
either from me building or driving into the area. In another embodiment the 
vehicle may park in a restricted zone where such parking is not allowed m 
specific hours or where parking or sending is reacted for a short period of rime 
The following operative conclusions are achieved by the suitable execution of 
sets of computer instructions embedded within a specificelly developed computer 
program. The program la operative in the analysis of a sequence of video tmages 
or other captured data received from me cameras covering a security-sensrave 
area (referred herein under to as the video scene). An illegal parking event is 
identified by the program.when the following sequence of sub-events takes place 
, and detected by the execution of the program: a) an object enters the video scene 
(76) The system identifies the object and in accordance with size and shape is 
assumed to be a vehicle occupied by one or more individuals, b) The object has 
subsequently to entering the restricted zone stopped moving (78). It is assumed 
ft* the vehicle has stopped, c) The dimensions of the objeet are above a pre- 
5 defined dimension parameter value (80). The dimensions of the object are 
ehecked in order to distinguish between vehicles and pedestrians that may enter 
the same area and stop therein (for example, sitting down on a bench). The size of 
fte object is also important to determine the direction of movement of the object 
Thus, objects growing bigger are moving in the direction of the capturing dev.ce 
while objects whose size is reduced over time are assumed to be movmg away 
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tao to capturing device. Certain predefined parameters as «o size may be pre- . 
pegged into the system in association with a specific background unage^ d) 

motion for a pre-defined period (82). As desenbed m 
association with Figs. 3A. 3B me system may detect whether an occupant of to 
5 object has left to object and is now about to leave to wider area around the 
video scene. Foliowing to identification of to previous sub-events, referred to as 
specific video scene characteristics, to event will be identified by to system as a 
situation in which an UlegaUy parked vehicle was left in the secunty-sensmve 
^ unattended. Titus, to parked vehicle will be considered as a suspicions 
10 object Another even, which is recognized as suspicious, is where me ; vehicle , . 
m0 ving in an unpredicted direction or if an object, such as a person, . leavmg the 
vehicle and moving to an unpredictable direction. An predicted direction cen be 
predefined and an unpredicted direction is to direction opposite or a direction 
wbich does not match a predefined direction of flow of persons or vehicles. For 
15 example, in an ahport the sidewalk where persons disembark ftom vemcles can 
be defined as a predicted direction and tire side opposite the sidewalk and across 
.he Janes of travel can be denned as the unpredicted direction. Thus, if persons 
disembarking vehicles at me airport leave the vehicle unattended (stationary for a 
predefined period of time) and walk or run not towards the airport, but rather to 
20 the opposite me system identifies a suspicious event In chy centers to same can 
be applied near bus depots or train stations and also in retell estimlishments 
monitoring areas not for public access. 

Another parameter, which can be viewed, is to speed of the object 
25 Speeding away ftom me vehicle can be an additional indicator to, a suspicious 
event is taking place. The parked vehicle may also be regarded as suspicious tf * 
is parked in to restricted zone more man a predefined period of tune. 
Consequently, to proposed system may generate, display and/or distribute an 
alarm indication. Alternatively, if to occupants of tire vehicle did not leave the 
30 vehicle bm still wait in the vehicle an alert can be raised, assuming a person is 
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W aiting in toe vehicle in suspicious circumstances or parking illegally, b to 
scenario too the parked vehicle will be considered a suspicious object 
Consequently, toe proposed system may generate, display and/or disnibute an 
aiarm indication. Once an alarm is raised toe officer reviewing toe momtored 
scene may request toe system to provide a playback so as to identify toe objects to 
,»estion. Once playback resumes toe officer may tap on a touch active screen 
I select toe image by otoer means such as a mouse, a keyboard, a Ugh, pen and 
the tike) and toe system may play back toe history of video captored m 
association witothe relevant object or objects. If a seeond object such as a person 
disembarked toe vehicle toe officer may tap toe object and rerpest a follow - V 
payback associated just with toat person. The playback or play forward feature 
atiows toe officer to make a real time determination as to toe object nature 
including information stored in toe database (such as parameter association wto 
toe object) and determine toe next action to be taken. 
, toanotoerentoodunentanalertmayberaiaedassoonasanobjeetm 

toe size of a vehicle as determined by the relative size of the object as predefined 
to toe system enters a restricted lane. The application concerning restricted lanes 
.nay cheek toe size of toe vehicles in such lanes as bus lanes wherein only buses 
( w bich are larger than vehicles) are allowed. If the object is a vehicle, i.e. smaUer 
, flum .bus, an alert may be raised The system may identify toe vehicle and latere 
ticket may be issued to toe owner of the vehicle. Ms applicntion is extremely 
useful for policing restricted lanes without having a police unit on the scene. 

In another embodiment of the invention, a database of recognized 
vehicle plate numbers can be utilized to assist in toe off line investigation and 
25 associated identification of toe owner of a suspicious vehicle. The database can 
aiso be used to determine whether toe number of me license plate is stolen or 
belongs to a suspect on a pre-suppUed list 

Referring now to Fig. 4B the exemplary control parameters of toe city 
center application are shown. In order to set up the illegal parking event detection 
application, toe user is provided with toe capability of defining toe following 
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parameters: a) area or areas wiflun the scaled zone wherein the ***** 
Lrch for suspect objects (84), b) the dimensional tts of the deteoted object 
(86). The minimal dimensional values provided in order <o limit the W e of obj ect 
as a vehicle, and e) a time out value (88) that is me amount of time mat may.pass 
. between the poittt-in-time at whieh the object popped moving and me po,n«-.n- 
time where an alarm will be generated. 

Note should be taken that the above^escribed steps for the detection 
of an illegally parked vehicle andthe associated control parameters are exemplary 
only Diverse olher sequences of steps and different parameters could be used m 
order to achieve the inherent objectives of the present invention. 

Referring now to Fig. 5 which is a flowchart describing the operation 
of toe proposed memod. The fundamental logical flow of the method is 
substantially similar for both the first preferred embodiment concerning the 
unattended baggage detection and for the second preferred embodiment 
5 concerning die detection of an illegally parked vehicle. At step 93 a set of 
suspicious behavior patterns are defined. At step 95 a set of appficatton and 
control parameters are defined. Both steps 93 and 95 are performed offline The 
behavior patterns are implemented in die DSA application 46 of Fig. Uwhfle the 
control parameters are stored in the application and setup parameter file 46 of 
20 Fig 1 At step 92 video data of a pre-defineu video scene is acquired by one or 
more video cameras. At step 94 the video stream is transmitted to the video 
capture unit and at step 96 the video data is sampled. Optionally the video data > 
recorded, compressed and archived on auxiliary storage devices, such as disks 
and/or magnetic rapes (step 98). Simultaneously the video input is transferred to 
25 the video analysis component The object detection unit analyzes the video date 
(step 100) and activates a specific DSA application 100. In the first preferred 
embodiment the DSA application 100 is the unattended luggage detection wbfie 
in me second preferred embodiment the DSA application 100 is the detection of 
an illegally parked vehicle. Another DSA application 100 is a lost object 
,0 prevention application. Other applications are evident from me description 
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provided above in association wnh Fig. 3A, 3B, 4A, and 4B. M step 1W « 
determined whetiter, in accordance wim the specific appbcation «d *. 
associated parameters, inclnding optiona. databaae parameters, an alarm stete^a 
^ by the video analysis .tit. If no alarm state was identified titen ptognun 
«,„ tt ol returns to me application 102. men an alarm state was rtnsed at step 106 
to a,arm even, ia insetted into the event database, at step 108 a 
ending alarm state deteils is sent to Otaphic Use, interface (GUT, contiol 
appUcation, a, step 110 an optional tnessage including alarm state-specffi detinfc 
Tsent to the optional alarm drslribution system to be distiibuted to a set of pre- 
defined monitor devices, such as wireless, personal date assistance devtces, 
pagers, telephones, e-mail and me like. 

The GUI contiol application 108 prompts the user for a amiable 
response concerning me alarm or optionally presenta me user in real time win, me 
video data sen, by the camera the omput of which generated the alarm. The alarm 
can be provided as text or pop up window onthe screen of the operator, as e-mad 
sent to an officer, SMS message sent to a cellular phone, an automated telephone 
eall to an officer, a text pager message, pictures or video steenm sen. me officer s 
portable device or hand held device, or send via a dry contec, to generate a snen 
or an audio or visual indication and me like. The message could be provtded to 
, „„e or many persons or to specific persons associated wim the specific even, or 
atom. The suspicions objec, on *e video images ia emphasized in a graph* 
m anner. such as encircling me objec, in a circle-like or oval graphic element ma, 
is overlaid on the video image. Other information concerning me object such as 
the objec ite size, speed, direction of movement range from camera, if identified 
, 5 and me like, will appear next to me objeofa image or in another location on me 
' screen. If the optional recording and archiving unit and tire associated vrdeo 
archive files are implemented on toe system men me user ia provrded wth the 
option of video date re-play. When the optional alarm distiibution component . 
^lamented onthe system, tire alarm message will be appropriately msurbuted 
30 to a set of pre-defined and suitable pre-configured locations. 
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Objects monitored by surveillance systems may move in unpredicated 
directions. In For example, in an airport surveiUance scene a person may arrive to 
the scene with a suitcase, enter the terminal building, leave the suitcase near the 

5 entrance of a terminal, and then leave the terminal. In another similar example, a 
first object (a vehicle) may arrive to the entrance of a terminal, a person (second 
object) may exit the vehicle, walk away in a direction opposite to the termmal, 
thus leaving the scene. In order to recognize patterns of unpredictable behavior a 
set of pre-defined rules could be implemented. These rules assist the system m 

10 capturing unpredictable behavior patterns taking place within the scenes 

monitored by the system 

The present system collects and saves additional information relating 
to each object An initial analysis is performed in connection with each object 
Apart from the circle-like shape and location of the object the system attempts to 
15 identify whether the object is a person or an manimate object. In addition, the 
object will collect and save object parameters such as the object-normalized S1 ze, 
distance from camera, color histogram. If the person is a person a face recognition 
algorithm is activated to try and determine whether the person is recognized. 
Recognized persons can be those persons that have been previously identified in 
20 other objects or may be faces that are provided to Ihe system, such as from law 
enforcement agencies or that are previously scanned by the employer. Other 
parameters may also be associated with the object such as name, other capturing 
devices, speed and the like. . 

When sufficient computing power is available, the system would also 
25 perform in real time a suitable analysis of the object in order to create associated 
search parameters, such as, for example, color histogram and other search 
parameters mentioned above and to immediately alert officers if the analysis leads 
to predetermined alarm status, such as when the a particular face is recognized 
which is a wanted person or a person not allowed or recognized in a restricted 
30 zone. In addition, in on line mode the system can identify more than one 
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parameters, such as a non-recognized face in a restricted zone and speaking m a 
foreign language, or a person not wearing a particular identifying mark (such as a 
hat or a shirt in a particular color) and me person is exiting a vehicle. 

The proposed system and method provide real-time and off-line 
processing of suspicious events. For example, when a vehicle arrives at a terminal 
of an airport or train station and a person leaves me vehicle to a direction opposite 
Ihe terminal, the present system and method will automatically alert me user. 
Such suspicious behavioral patterns are predetermined and me present system and 
method analyzes events to detect such events. The present system and method is 
further capable of identifying a set of linked events associated with the same 
object An object can be defined as any detected object mat continues to move 
within the captured scene. An event is defined as a series of frames capturing a 
scene and objects mere within. The event can be associated with a particular 
capturing device. Linked events to the same object relate to a single object in the 
same area throughout , the surveillance period whether captured by one or more 
cameras and appearing in one or more events. The system will track (either upon 
request or automatically) an object through one or more events. The present 
system and method also provide the ability to associate a retrieved event or object 
with unique parameters of such an object, in addition to the object oval 
3 characteristics and location. Such would include, for example face recognition, 
color of clothes through the use of a histogram color. The difference between the 
color of the clothes and the color of the shirt of an object, color per zone m the 
object, such as the color of a hair, normalized size subject to the distance from the 
camera, and normalized shape of objects such as the size of a suitcase. The use of 
, 5 object associated parameters in addition to the object's shape and position enable 
the post event data base search of an object according to the parameters to quickly 
obtain the event or events associated with the object or other objects associated 
with the object Such parameters also enable the user of the present invention to 
investigate and request the system to identify a particular object or event. This 
30 enables a better retrieval of the events and objects. The system may also, in real 
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time, associate the parameters with objects and perform rule checking to 
determine if the objects comply with rules that are permitted in the scene, such as 
objects are not left unattended, objects move is specific directions, objects do not 
depart from other objects in specific locations, and the like. 

The additional embodiments of the present system and method will 
now be readily apparent to person skilled in the art. Such can include crowd 
control, people counting, an offline and online investigation took based on the 
events stored in the database, assisting in locating lost luggage (lost prevention) 
and restricting access of persons or vehicles to certain zones. The applications are 
both for city centers, airports, secure locations, hospitals and the like. 

It will be appreciated by persons skilled in the art that the present 
invention is not limited to what has been particularly shown and described 
hereinabove. Rather the scope of the present invention is defined only by the 
claims, which follow. 
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CLAIMS 

WHAT IS CLAIMED IS: 

1 . A method for analyzing video data, comprising: 
5 receiving a video frame; 

comparing said video frame to at least one background 
reference frame to locate at least one difference; 

locating a plurality of objects to form a plurality of 

marked objects; and 
10 . determining a behavior pattern for at least one object 

according to said at least one difference, wherein said 
. behavior pattern is defined according at least one scene 
characteristic. 

15 2. The method of claim 1, further comprising producing at 

least one updated background reference frame. 

3. The method of claim 1, wherein deterniining said at least 
one difference is performed by creating a difference frame 

20 between said video frame and said at least one background 

reference frame. 

4. The method of claim 1, further comprising finding at least 
one new object when deterniining said at least one 

25 difference. 

5. The method of claim 1, further comprising issuing an alarm 
according to said behavior pattern. 
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6. The method as claimed in claim 1 wherein a pre-defined 
pattern of suspicious behavior comprises an object 
presenting unpredictable behavior. 

7. A system for aiialyzing video data comprising a plurality of 
video frames, the system comprising: 

a video frame preprocessing layer for determining a 
difference between a plurality of video frames; 

an object clustering layer for detecting a plurality 
of objects according to said difference; and 

an application layer for characterizing said plurality of 
objects according to at least one scene characteristic. 

8. The system of claim 7, wherein said difference is 
determined between a video frame and at least one 
reference frame, the system further comprising a 
background refreshing layer for preparing at least one 
updated reference frame according to the said difference. 

9. The system of claim 8, wherein said at least one scene 
characteristic defines a behavior pattern for at least one 
object, such that if said at least one object exhibits said 
behavior pattern, said at least one scene characteristic is 
detected. 

10. The system of claim 9, wherein if said at least one scene 
characteristic is detected, an alarm is generated. 

11. The system of claim 9, wherein said at least one scene 
characteristic further comprises at least one parameter for 
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determining if said at least one object exhibits said behavior 
pattern. 



5 12. A system for detecting a vehicle remaining in a restricted 

zone for at least a rninimum period of time, comprising: 

a video content analysis module for analyzing video data 
of the restricted zone, said video content analysis module 
further comprising an object tracking component; and 

10 an application layer for receiving data from said video 

content analysis module and for detecting a vehicle 
remaining in the restricted zone for at least the minimum 
period of time, and said application layer generating an 
alarm upon detection. 
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13. A system for detecting unattended luggage, bag or any 
unattended object in an area, comprising: 
a video content analysis module for analyzing video data of 
the area, said video content analysis module further 
comprising an object Hacking component; and an 
application layer for receiving data from said video content 
analysis module and for detecting an unattended object, 
wherein said unattended object has not been attended in the 
area for more then a predefined period of time. 

14. A surveillance system for 1he detection of an alarm 
situation, the system comprising the elements of : 

a video analysis unit for analyzing video data representing 
images of a monitored area, the video analysis unit 
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comprising an object tracking module to track the 
movements and the location of a video obj ect; 

a detection, surveillance and alarm application for 
receiving video data analysis results from the video analysis 
unit, for identifying an alarm situation and to generate an 
alarm signal; 

an events database to hold video objects, video object 
parameters and events identified by the application. 

The system of claim 14 further comprises the elements of: 

an application driver to control the detection, surveillance 
and alarm application; 

a database handler to access, to update and to read the 
events database; 
a user interface component to communicate with a user of 

the system; 

an application setup and control component to define the 
control parameters of the application; 

an application setup parameters table to store the control 
parameters of the application. 

16- The system as claimed in claim 14 further comprises the 
elements of: 

a video da t a recording and compression unit to record and 
25 compress video data representing images of a monitored 

area; 

a video archive file to hold the recorded and compressed 
video data representing images of the a monitored area; 
an alarm distribution unit to distribute the alarm signal 
30 representing an alarm situation. 

30 
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17. Hie system as claimed in claim 14 further comprises the 
elements of: 

at least one video camera to obtain the images of a 
5 monitored area; 

at least one a video capture component to capture video 
data representative of the images of Ihe monitored area; 

at least one video transfer component to transfer the 
captured video data to the video analysis unit and the 
10 recording compressing and archiving unit; 

at least one computing and storage device. 

18. The system as claimed in claim 14 wherein the object 
tracking module comprises the elements of: 
15 a video frame preprocessing layer for determining the 

difference between at least two video frames; 

an objects clustering layer for detecting at least one 
objects in accordance with the determined difference; 
a scene characterization layer for characterizing the a least 
20 one object according to least one characteristic of a scene; 

a background refreshing layer for preparing at least 
one updated reference according to the determined 
difference. 
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19. The system as claimed in claim 14 wherein the detection 
surveillance and alarm application is operative in the 
detection of at least one unattended object in the monitored 



area. 
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20. Hie system as claimed in claim 17 wherein any of the at 
least one video camera; the at least one video capturing 
component; the at least one video transfer component and 
the at least one computing and storage device can be 
separated and can be located in different locations. 

21. The system as claimed in claim 17 wherein the interface 
between the at least one video camera; the at least one video 
capturing component; the at least one video transfer 
component and the at least one computing and storage 
device is a local or wide area network or a packed-based or 
cellular or radio frequency or micro wave or satellite 
network 



15 

. 22. The system as claimed in claim 19 wherein the at least one 
unattended object is a luggage left in an airport terminal for 
a pre-determined period. 

20 23. The system as claimed in claim 22 Ihe at least one 

unattended object is a vehicle parking in a restricted zone 
for a pre-defined period. 

24. The system as claimed in claim 14 wherein the detection 
25 surveillance and alarm application is operative in the 

detection of an unpredicted object movement 

25. The system as claimed in claim 14 wherein the analysis is 
also performed on audio data or thermal imaging data or 
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radio frequency data associated with the video data or the 
video object in synchronization with the video data. 

< 

26. The system as claimed in claim 17 wherein the video 
capture component captures audio or thermal information or 
radio frequency information in synchronization with the 
video data. 



27. A surveillance method for the detection of an alarm 
situation, the surveillance to be performed on at least one 
monitored scene having at least one camera, the method 
comprising the steps of: 

obtaining video data from the at least one camera 
representing images of a the at least one monitored scene; 

analyzing the obtained video data representing images 
of the at least one object within the at least one monitored 
scene, the analyzing step comprising of identifying the at 
least one object within the video data; and 

inserting the identified at least one object and the at least 
one event into an event database. 

28. The method of claim 27 further comprising the steps of: 

retrieving at least one of the object associated with at 

least one event; 

according to user instruction displaying the video 
event associated with the retrieved at least one object 

29. The method of claim 27 further comprising the steps of: 
retrieving at least two events; 
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associating according to parameters of the at least one 
object, the at least one object with the at least two events. 

30. The method of claim 27 further comprising the steps of: 
debriefing the at least one object associated with the at 

least one event to identify the pattern of behavior or 
movement of the at least one object within the at least one 
scene within a predefined period of time. 

31. The method of claim 27 further comprising the steps of: 
pre-defining patterns of suspicious behavior; and 
pre-defining control parameters. 

32. The method of claim 27 further comprising the steps of: 
recognizing an alarm situation according to the pre- 
defined patterns of suspicious behavior; and 

generating an alarm signal associated with the recognized 
alarm situation. 

33. The method as claimed in 27 further comprises the steps of. 
implementing patterns of suspicious behavior 
introducing pre-defined control parameters; 
recording, compressing and archiving the obtained 
video data; 

distributing the alarm signal representing an alarm 
situation across a pre-defined range of user devices. 

34. The method as claimed in claim 27 wherein a pre-defined 
pattern of suspicious behavior comprises: 
an object entering a monitored scene; 
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the object separating into a first distinct object and a second 
distinct object in the monitored scene; 

the first distinct object remaining in the monitored scene 
without movement for a pre-defined period; and 
5 the second distinct object leaving the monitored scene, 

35. The method of claimed in claim 34 wherein the pre-defined 
pattern of suspicions behavior comprises: 
an object entering the monitored scene; 
1 o the obj ect ceasing its movement; 

the size of the object is recognized as being above a 
pre-defined parameter value; and 

the object remaining immobile for a period recognized 
as being above a pre-defined parameter value. 
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36. The method of claim 35 further comprising identifying 
information associated with the object for the purpose of 
identifying the at least one object 
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